Papers with downstream accuracy
Pretraining Without Attention (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that state-space models (SSMs) outperform standard and deep learning for long-range sequence modeling. |
| Approach: | They propose a model that combines SSM layers with a multiplicative gating architecture that has been effective in simplified sequence modeling architectures. |
| Outcome: | The proposed model outperforms standard and standard sequence modeling architectures on speech generation and the long range arena benchmarks. |
GEAR: Augmenting Language Models with Generalizable and Efficient Tool Resolution (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent work on Augmented Language Models (LLMs) over-rely on task-specific demonstrations that limits their generalizability and computational cost. |
| Approach: | They propose a query-tool grounding algorithm that is generalizable to various tasks . they delegate tool grounding and execution to small language models and LLMs . |
| Outcome: | The proposed algorithm outperforms baselines on 14 datasets and shows it can be generalized to different tasks. |
BanglaSTEM: A Parallel Corpus and Term-Weighted Evaluation for Technical Bangla-English Translation (2026.acl-srw)
Copied to clipboard
| Challenge: | Large language models excel at technical problem solving in English but struggle when questions are posed in Bangla. |
| Approach: | They propose a dataset of 5,000 Bangla-English sentence pairs to align technical terms . they use OCR to extract matching passages from bilingual textbooks . |
| Outcome: | The proposed pipeline extracts matching passages from bilingual textbooks and uses them to align sentences and mark technical terms. |
From Pseudo-Balancing to True Specialization: Memory-Aware Routing for Mixture-of-Experts (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to optimize expert-centered load balancing fail to account for pseudo-balance phenomenon . severe knowledge overlap among experts leads to redundant representations and inefficient parameter utilization . |
| Approach: | They propose a method that prioritizes expert utilization over semantic alignment . they use memory-aware routing to ensure expert load balancing is consistent . |
| Outcome: | Experimental results show that MAR improves expert specialization by 35% and accuracy by 2%-25% . MAR matches baseline performance with only half the experts . |
Can we Retrieve Everything All at Once? ARM: An Alignment-Oriented LLM-based Retrieval Method (2025.acl-long)
Copied to clipboard
| Challenge: | Existing RAG solutions address the alignment problem in a limited manner . ARM explores relationships among data objects, enabling a retrieve-all-at-once solution for complex queries . |
| Approach: | Experimental results show that ARM improves alignment of open-domain questions with available data . ARM explores relationships among data objects, enabling a retrieve-all-at-once solution for complex queries. |
| Outcome: | Experimental results show that ARM outperforms existing RAG methods on complex open-domain questions. |
From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models (2026.acl-long)
Copied to clipboard
Ziyan Wang, Enmao Diao, Qi Le, Pu Wang, Minwoo Lee, Shu-ping Yeh, Evgeny Stupachenko, Hao Feng, Li Yang
| Challenge: | Structured pruning is a practical approach to deploying large language models (LLMs) but it fails to capitalize on modest task-specific calibration signals, causing limited downstream gains. |
| Approach: | They propose a method that removes attention heads and MLP channels using loss-based important scores . they use perplexity for language modeling and a margin-based objective for decision-style tasks . |
| Outcome: | The proposed method lowers perplexity and improves accuracy at higher sparsity . it also stabilizes accuracy and mitigates perxity collapse without fine-tuning . |